A Convergent Reinforcement Learning Algorithm in the Continuous Case Based on a Finite Difference Method

The problem of computing a good approximat ion of the value function, which is essential because this provides the optimal control, is a difficult task in the continuous case. Indeed, as it has been pointed out by several authors, the use of parameterized functions such as neural networks for approximating the value function may produce very bad results and even diverge. In fact, we show that classical algorithms, like Q-learning, used w i th a simple look-up table bui l t on a regular grid, may fail to converge. The main reason is that the discretizat ion of the state space implies a lost of the Markov property even for deterministic continuous processes.